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Epidemiology Faces Its Limits 

The search for subtle links between diet, lifestyle, or environmental factors and disease is 
an unending source of fear—but often yields tittle certainty 


The news about health 
rules comes thick and fast 
these days, and it seems al¬ 
most constitutionally con¬ 
tradictory. In January of 
last year, for instance, a 
Swedish study found a sig¬ 
nificant association be¬ 
tween residential radon 
exposure and lung cancer. 

A Canadian study did not. 

Three months later, it was 
pesticide residues, The 
Journal of iht National Can ■ 
cer Insatuif published a 
study in April reporting— 
contrary to previous, less 
powerful studies—that the 
presence of DDT metabo¬ 
lites in the bloodstream 
seemed to have no effect 
on the risk of breast can¬ 
cer. In October, it was 
abortions and breast cancer. Maybe yes. 
Maybe no. In January of this year it was 
electromagnetic fields (EMF) from power 
lines. This time a study of electric utility 
workers in the Unired States suggested a pos¬ 
sible link between EMF and brain cancer 
but—contrary to a study a year ago in 
Canada and France—no link between EMF 
and leukemia. 

These are not isolated examples of the 
conflicting nature of epidemiologic studies: 
they're just the latest to hit the newspapers. 
Over the years, such studies have come up 
with a intnd-nunlbmg atTay of potential dis¬ 
ease-causing agents, from halt dyes (lym¬ 
phomas, myelomas, and leukemia) to coffee 
(pancreatic cancer and heart disease) to oral 
contraceptives and other hormone treat¬ 
ments (virtually every disorder known ro 
woman), The pendulum swings back and 
forth, subjecting rhe public to an "epidemic 
of anxiety," as Lewis Thomas put it over a 
decade ago. Indeed, Iasi July, the New England 
Journal of Methane [NEJi M) published nn edi¬ 
torial by editors Marcia Angel! and Jerome 
Kassircr asking the puhy question. "What 
Should the Public Believe?" Health-con¬ 
scious Americans, wrote Angell and (Cas¬ 
sirer, increasingly find themselves beset by 
contradictory advice. Ho sooner do they 
learn the results of one research study than 
they hear oi one with the opposite message." 

fCassircr and Angcll place responsibility 
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Anxiety epidemic. Protesting risks 
(Hat may—or may not—be real. 


i? on the press for its report- 
£ ing of epidemiology, and 
" even on the public “for its 
| unrealistic expectations" of 
<e what modem medical re- 
t search can do for their 
| health. But many epidemi¬ 
ologists interviewed by Sci¬ 
ence say the problem also 
lies with the very nacure of 
epidemiologic studies—in 
particular those that cry to 
isolate causes of noninfec- 
tious disease, known vari¬ 
ously as “observational” or 
“risk-factor" or "environ¬ 
mental" epidemiology. 

The predicament of 
these studies is a simple one: 
Over the past 50 years, epi¬ 
demiologists have succeeded 
in identifying the more con¬ 
spicuous determinants of 
noninfectious diseases—smoking, for in¬ 
stance. which can increase the nsk of develop¬ 
ing iung cancer by as much as 3000%. How 
they are left to search for subtler links be¬ 
tween diseases and environmental causes or 
lifestyles. And that leads to the Catch-22 of 
modem epidemiology. 

On the one hand, these subtle risks—say, 
the 30% increase in the risk 
of breast cancer from alco¬ 
hol consumption that some 
studies suggest—may affect 
such a large segment of the 
population that they have 
potentially huge impacts on 
public health. On the other, 
many epidemiologists con¬ 
cede that their studies are so 
plagued with biases, unecr- 
and mcihodolugi- 
tliat thev 


ratlines 

cat weaknesses 
may be inherently incapable 
nf accurately discerning 
such weak associations As 
Michael Thun, the director 
of analytic epidemiology for 
the American Cancer Soci¬ 
ety, puts n, "With epidemi¬ 
ology you can tell a little 
thing from a big thing. 
What 's very hard to do is to 
tell a littie thing from noth¬ 
ing at all." Agrees Ken 



Rothman, editor of the journal Epidemiokigy: 
“We're pushing the edge of what can be done 
with epidemiology.” 

Wich epidemiology stretched to its limits 
or beyond, says Dimitrios Trichopoulos, 
head of the epidemiology department at the 
Harvard School of Public Health, studies 
will inevitably generate false positive and 
false negative results “with disturbing fre¬ 
quency." Most epidemiologists are aware of 
the problem, he adds, “and tend to avoid 
causal inferences on the basis of isolated 
studies or even groups of studies in the ab¬ 
sence of compelling biomedical evidence. 
However, exceptions do occur, and their fre¬ 
quency appears to be increasing." As 
Trichopoulos explains, “Objectively the 
problems are not more than they used to be. 
but the pressure is greater on the profession, 
and the number who practice ic is greater." 

As a result, journals today are full of stud¬ 
ies suggesting that a little risk is not nothing 
at all. The findings are often routed in press 
releases by the journals that publish them 
or by the researchers' insacucions, and news¬ 
papers and other media often report the 
claims uncritically (see box on p. 166). And 
so the anxiety pendulum swings ac an ever 
more diitying rate. "We are fast becoming a 
nuisance to society," says Trichopouios. 
“People don'r rake us seriously anymore, and 
when they do take us seri¬ 
ously, we may unintention¬ 
al ly do more harm than good." 
As a solution, epidemiolo¬ 
gists interviewed by Science 
could suggest only that the 
press become more skeptical 
of epidemiologic findings, 
that epidemiologists become 
more skeptical about their 
own findings—or borh. 


“Peopte don’t take 
us seriously... and 
when they do ... 
we may uninten¬ 
tionally do more 
harm than good.” 

—Dimitrios 
Trichopoulos 


An observational science 

What drives rhe epidem¬ 
iologic quest for risk factors is 
the strong circumstantial 
evidence that what we cat, 
drink, breathe, and so on are 
major factors in many devas¬ 
tating illnesses. Rales of 
heart disease, for example, 
have changed much faster 
over recent decades than can 
he explained by genetic 
changes, implicating dietarv 
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Sizing Up the 

la die history of epidemiology, on'y a dozen or*o environmental 
•gents have ever been repeatedly and strongly linked to human 
cancel, ays University of Alabama epidemiologist Philip Cole. 
Among them are cigarette smoke, alcohol, ionizing radiation, a 
few drugs, a handful of occupational carcinogens, such as asbestos, 
and perhaps three viruses—hcpatitis-B virus, human T cell leuke¬ 
mia virus; and human papillomavirus. But every year, epidem¬ 
iologic papers are published by the journal-load; many of them 
reporting new potential causes of cancer in the environment. 

Most arc the product of observational epidemiology, in which 
researches try to compare che lives of people suffering from a 
disease with those of healthy controls. Even its practitioners 
admit this effort is plagued by biases and confounding factors (see 
main text); As a result, most epidemiologists interviewed by 
Science said they would not cake seriously a smgfe Study reporting 
a new potential cause of cancer unless it reported that exposure to 
the agent in question increased a person's risk by at least a factor 
of3—which is to say it carries a risk ratio of 3. Even then, they say, 
skepticism is in order unless the study was very large and ex¬ 
tremely well done and biological data support the hypothesized 
link. Sander Greenland, a University of California, Los Angeles, 
epidemiologist, says a study reporting a twofold increased risk 
might then be worth taking seriously—“but not thac seriously." 

Few of the entries in the following list of potential cancer risks, 
reported in the journals and picked up in the popular press over 
the past 8 years, have come close to fulfilling those criteria. Are 
these dangers real? As che saying goes, you be the judge. 

-G.T. 

Hhjh-ehoiesteroi diet—risk ratio (rr) 1.65 for rectal cancer in 
men (January 1987) 

Eating yoqurt fit least once a month—ft 2 for ovarian cancer 
(July 1989) 

Smoking more than 100 cigarettes in a lifetime—rr 1.2 lot 
breast cancer (February 1990) 

High-fat diet— rr 2 for breast cancer (August 1 990) 

Lengthy occupational exposure to dioxin— rr 1.5 for ail cancers 
(January 1991) 

Douching once a week—rr 4 for cervical cancer (March 1991) 

Regular use of High-aicohof mouthwash —rr 1,5 for mouth 
cancer (June 1991) 


Cancer Risks 

Use of phenoxy herbicides on fawns—rr f.3 for malignant 
lymphoma in dogs (September 1991) 

Weighing 3,6 kilograms or more at birth—rr t.3 for breast 
cancer (October 1992) 

Vasectomy—rr 1.6 for prostate cancer (February 1993) 

Pesticide exposure. Indicated by high residues in blood—rr A 
for breast cancer (April 11993); contradicted 1 year later in a 
larger study with one of the same authors. 

Drinking more than 3.3 liters of fluid (particularly chlorinated 
tap water) a day—fr 2-4 for bladder cancer (July 1993) 

Experiencing psychological stress in the workplace—rr S.5 for 
colorectal cancer (September 1993) 

Diet high in saturated fat—rr 6 for lung cancer in nonsmoking 
women (December 1993) 

Eating more than 20 grams of processed meats (Le, bologna) 
a day—rr 1.72 for colon cancer (February 1994) 

Eating red meat five or more times a week—rr Q.5 for colon 
cancer (February 1994) 

Occupational exposure to electromagnetic fields—rr 1.38 for 
breast cancer (June 1994) 

Smoking two packs of cigarettes a day—rr 1.74 for fatal breast 
cancer (July 1994) 

Eating red meat twice a day—rr 2 for breast cancer (July 1994| 

Regular cigarette smoking—rr 1.7 for pancreatic cancer (Octo¬ 
ber 1994) 

Ever having used a sun lamp—rr 1.3 for melanoma (November 
1994) 

Abortion—rr 1.5 for breast cancer (November 1994) 

Having shorter or longer than average menstrual cycles—rr 2 
for breast cancer (December 1994) 

Obesity in men (the heaviest 25% of those in the study)—rr 3 
for esophageal cancer (January 1995) 

Consuming olive oil only once a day or less—rr 1.25 for breast 
Cancer (January 1995) 


and environmental causes. And the fact that 
no single cancer affects every population at 
the same rate suggests that factors cxrcrnul rci 
the human body cause 70% to 90% of all 
cancers. In other words, says Richard Pero, 
an Oxford University epidemiologist, “there 
arc ways in which human beings can live 
whereby those cancers would not arise." 
Only a few of rhesc environmental factors 
are known-—cigarette smoke for lung cancer, 
for example, or sunlight for shin cancer— 
and epidemiology seems to provide rhe best 
shot at identifying the others. 

The most powerful cool for doing so is rhe 
randomized trial, which is the standard for 
studies of new drugs and other medical re¬ 


search: Assign subjects at random to test and 
Control nmups, alter the exposure of rhe test 
group to the suspected risk factor, and follow 
both groups to learn rhe outcome. Often, 
both tlu- experimenters and the subjects are 
"blinded*'—kin a ware who is in rhe tot urotip 
and win) is a control. Bur randomized trials 
would he prohibitively slow and expensive 
for most risk factors, because they can rake 
years or decades ro show an effect and hun¬ 
dreds of thousands <>f individuals inav need to 
he followed to detect enough case* of the 
disease for the results to lie significant- And 
randomly subjecting thousands of healthy 
people to pollutants or other p<>s<ih}e car¬ 
cinogens raises obvious ethical problems. 
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IVcuise the experimental approach isuft- 
1 units for much of epidemiology, researchers 
resort ro observational approaches In case- 
comrol studies, for example, they select a 
eroup of individuals afflicted with a particu¬ 
lar disorder, then identify a control group 
free of rile disorder and compare the two, 
looking for differences in hfesrvle, diet, or 
some environ mental factor. Potentially 
more reliable, bur also much more costly, are 
cohort studies, in which researchers take a 
large population—as many a* 100.000—and 
question the subjects m derail about iheir 
babtrs and environment. They then follow 
the entire population for years nr decades to 
see who gers sick and who doesn’t, what dts- 
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Press Coverage: Leaving Out the Big Picture 


In the past 2 years, thorough readers of the Los Angeles Times 
would have learned about an extraordinary range of potential 
cancer causes. Among these putative hazards of modem life are 
hot dogs, breast implants, dioxin, Stress, asbestos, allergy drugs, 
gas halts, living in Orange County, tubal ligation, sunscreen, 
Asian food, pesticides, vasectomy, liquor, working in restaurants, 
Retin-A, vegetables, dietary fat, delayed child-bearing, impuri¬ 
ties in meat, and lesbianism. This litany of fear was accompanied 
by a similar, although shorter, series of reports on dietary habits 
and lifestyles that may reduce cancer risk. Parallel coverage ap¬ 
peared in other newspapers and magazines and on television. To 
many scientists, though, the media would do well to curb its 
appetite for such news. 

The problem, many researchers say, is chat journalists often 
misunderstand the context ofthe research. Because of the limita¬ 
tions of risk-factor epidemiology, most individual studies cannot 
produce authoritative findings (see main text). “Articles pub¬ 
lished in medical journals are often misconstrued by the lay press 
to be more definite than they really are," says Latry Freedman, a 
bioscatistician at the National 
Cancer Institute. "Broccoli pre¬ 
vents cancer, garlic prevents 
cancer—all these things do ap¬ 
pear in the literature. But epide¬ 
miologists understand very well 
that these studies are far from 
definitive. It’s only when a body 
of evidence exists over many, many studies 
that epidemiologists should really get seri¬ 
ous about giving the public advice," 

Instead of presenting surveys of the big, 
evolving picture, he and others say, the me¬ 
dia tend to report each new study in isolation, 
as a new breakthrough. Such reporting, some 
scientists say, is encouraged by press releases put out by 
journals and researchers' institutions. Bur whoever is iu 
blame, saysNocI Weiss, an epidemiologist at the University 
of Washington. Seattle, the result is “just too many false 
alarms. When wc do have a serious message. 1 fear it won't 
be heeded because of the large number of false messages ” 

One example is an item from Time magazine's "Health 
Watch,” which tersely summarizes recent research Published 
last January, the item read, in irs entirety, "Olive orl seems to do 
more than make food taste gixxi. Research indicates that women 
who consume nil ve oil more than once a day reduce their risk of 
breast cancer 25% compared with women who don’t.” Time 
didn t mention that the risk reduction is smaller than many 
epidemiologists think can be reliably detected in an observa¬ 
tional study. Nor did u point out that the study—apparently a 
Study of 1750 Spanish women reporced several weeks earlier in 
the hucmauunal Journal of Cancer —is in conflict with many other 
studies suggesting that dietary tars may raise rarher than lower the 
risk of breast cancer. Although the overall fat-breast cancer I ink 
is disputed, and olive oil may pose less cardiovascular risk [han 
other forms of far, few epidemiologists would interpret these 
findings as indicating thac women should “consume olive oil 
more than once a day." 

In their proclivity for "news,” newspaper and television re¬ 
porters not only single out weak studies; they may focus on the 
one positive result in a sea of negative data. That was the case 
with coverage of* two big studies on occupational exposure to 


electromagnetic fields (EMF) that appeared recent 1 y in che Amer¬ 
ican Journal o/Epidemiology. The first study, of223,000French and 
Canadian electric utility workers, found no link between EMF 
and 25 of the 27 varieties of cancer in the study; the exceptions, 
two rare types of leukemia, had a weak and inconsistent positive 
association with EMF. Yet the Waif Street Journal reported the 
study last spring under the headline, “Magnetic Fields Linked to 
Leukemia.*' 

Early this year the American Journal of Epidemiology published 
the second study, on 139,000 workers at five U.S1 utilities. It 
found no association between exposure to EMF and 17 of 18 types 
of cancer, including che leukemias linked to EMF by the first 
study. The sole exceptions were eye and brain cancers—condi¬ 
tions thac had shown no link to EMF in the first study. Yec the 
headline of che Wall Street Journal article that reported the second 
study was “Link Becween EMF, Brain Cancer Is Suggested by 
Study at 5 Utilities." Says Jerry Bishop, who wrote one of the Wall 
Street Journal articles, “People are not interested in what diseases 

[a risk factorl doesn't cause, but what it mighccause_We've had 

this argument with scientists many 
times over the past few years." 

In October, the New York Times 
provided another example, when it 
reported on a study in the journal of 
the Naiionai Cancer Insotuts (JNCf) 
from the Fted Hutchinson Cancer Re¬ 
search Center in Seattle that suggest¬ 
ed induced abortion might increase 
the risk of breast cancer by 50%. Al¬ 
though the article noted that 40 pre¬ 
vious studies of abortion and breast 
cancer had found no such correlation, 
tbe headline read “New Study Links 
Abortions and Increase in Breasc 
Cancer Risk." Inevitably, public at¬ 
tention was directed to a risk that is 
unlikely to be real. 

If there is "blame" for such cover¬ 
age, argues Lawrence Airman, author 
of the Times article, much of it belongs 
to scientific journals, “The JNCi sent 
out n big release routing chut study as if it were the biggest tiling 
since whatever," hr.says. "I don’t recall them telling us that it was 
only one of 40 studies and probably had little meaning." 

In Altman’s view, epidemiologists who complain about l 
press coverage are trying ru have it both ways. "Scientists sup- I 
posedly want us not to go outside the scientific process, but wait 
until findings have appeared in a pccr-rcviewcd professional ! 
journal. When we do that, ihcy apparently complain that we ] 
didn’t go outside tbe scientific process and say that a published 
report is meaningless." 

"Journalists do overemphasize individual studies, but they are 
often invited to do that [by medical journalsl." agrees Ross j 
Prentice, one of Weiss’s colleagues at the University of Washing¬ 
ton. "I’ve seen some of the press releases that journals and univer- ^ 
stties send to journalists. It’s a wonder sometimes that the report- ! 
mg is as good as it is." 

-Charles C. Mann 


Charles C Mam is the co-auihcrr. with Mark L. Plummer, of Noah's 
Choice; The future of Endangered Species. 
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c;»M-s ill try suffer from, and what factors might 
lx- different between diem. Either way, risk- 
facinr epidemiology is “a much duller scal¬ 
pel" than randomized uials, says Scott Zcger, 
a liiiutatistician at the Johns Hopkins School 
of Mental and Public Health. 

What blunts its edge ate systematic errors, 
known in the lingo as biases and confound¬ 
ing factors, “Bias and confounders are che 
plague upon the house of ep idem iology ."says 
Philip Cole, chair of epidemiology at the 
University of Alabama. They represent any¬ 
thing that might lead an epidemiologic scudy 
to come up with che wrong answer, to postu¬ 
late the existence of a causal association that 
does not exist or vice versa. 

Confounding factors are the hidden vari¬ 
ables in the populations being studied, which 
can easily generate an association that may 
he real but is not what the epidemiologist 
thinks it is. A ubiquitous example is cigarette 
smoking, which can confound any study 
looking, for instance, at the effectsof alcohol 
cm cancer. “Ic just so happens," explains 
Trichopoulos, "that people who drink also 
tend to smoke,” boosting their risk of cancer. 
Asa result, epidemiologists face the possibil¬ 
ity that any apparent cancer-alcohol link 
may be spurious. Smoking may also have 
confounded a study Trichopoulos himself 
co-authored linking coffee-drinking and 
pancreatic cancer—a finding that has not 
been replicated. The Study, published over a 
decade ago, corrected for smoking, which 
often accompanies heavy coffee drinking— 
hut only for smoking during the 5 years be¬ 
fore the cancer was d lagnosed. T richopou |os 
now says chat he and his colleagues might 
have done better to ask about smoking habits 
a full 20 years before diagnosis. 

Biases are problems within study designs 
themselves. The process of choosing an ap¬ 
propriate population of controls in a case- 
control study, for instance, can easily lead to 
an apparent difference between cases and 
controls that has nothing to do with what 
caused the disease, “it’s often not even theo¬ 
retically clear who the right comparison 
group is,” says Harvard epidemiologist 
Walter Willett. “And sometimes, even if yoti 
can design the study so that you have the 
theoretically correct comparison group, you 
usually don't getevcryhoJy willing to partici¬ 
pate, and the people who do participate in 
V'ur study will be different from the people 
n ho don't, often in health-related ways.” 

For example, Charles Pixilc of Boston 
University has spent several years analyzing 
the results and inerfmdulugy of a I98fi stujy 
of EMF and cancer, which found that expo¬ 
sure to relatively high EMF from power lines 
appeared to increase the risk of leukemia and 
brain cancer in children. David Savin of the 
University of North Carolina, the study's 
author,selected controlsfor that study with a 
common technique known as random digit 


dialing: Researchers take the phone numbers 
of their cases and randomly change the last 
four digits until they find a suitable control. 
Random digit dialing, however, seems to cre¬ 
ate "a pronounced bias toward the control 
group being deficient in persons of very low 
socioeconomic status," says Poole. Poor 
people, it seems, are either less likely to be 
home during the day to answer the phone, 
less likely to want to take part in a study, or 
less likely to have an answering machine and 
call the researchers back. 

Indeed, the North Carolina researchers 
reported chat their data showed that che risk 
of leukemia and brain cancer 
rises not just with exposure 
to EMF but also wich higher 
levels of breast-feeding, ma- 
cemal smoking, and traffic 
density, all of which are mar¬ 
kets for poverty, This sug¬ 
gests. says Poole, that the study 
group was poorer than the 
controls, and that some pov¬ 
erty-associated factor other 
than EMF could have re¬ 
sulted in the apparent in¬ 
crease in cancer risk. None¬ 
theless, the study is still cited 
as supporting the hypothesis 
that EMF causes childhood 
cancel, although even Savitz 
concedes that the random dig¬ 
it dialing problem is "a legiti¬ 
mate source of uncertainty." 

Even when such biases 
can be identified, their mag¬ 
nitude—and sometimes even theit direc¬ 
tion—can be nearly impossible to assess, 
David Thomas, for example,an epidemiolo¬ 
gist at the Fied Hutchinson Cancer Research 
Center m Seattle, points to studies analyzing 
the effect of Breast Self-Examination (BSE) 
on breast cancer mortality rates, which, he 
says, have yielded some "modesr suggestion 
that there might be a beneficial effect" from 
BSE. “You have to ask what motivates a 
woman to praec/ce BSE," says Thomas. 
"Maybe she has a strong family history of 
breast cancer. If so, she's more likely to get 
hreasr cancer. Tli.u would he an obvious 
bias." which could make BSE look less useful 
than it is. "Or maybe a wuman with a strong 
family history of breast cancer would lie 
nfr.iiJ in practice BSE. You have no way of 
predicting the direction of the bias. Si n 
would be very difficult to interpret your re¬ 
sults. You have to go to a randomised study to 
get a tellable answer." 

Tricks of memory 

Of all the biases that plague rhe epidemio¬ 
logic study of risk factors, the most pernicious 
is the difficulty of assessing exposure to a 
particular risk factor. Rothman. fur instance, 
calls it "a towering obstacle." When expo- 
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sure can be measured reliably, » subtle asso¬ 
ciation may be credible—as it is in the case of 
early childbirth and a lower risk of breast 
cancer. The reason is chat both cause and 
effect can be measured with some certainty, 
says Harvard epidemiologist Jamie Robins- 
“It's easy to know which people got breast 
cancer, and it’s easy to knowar what age they 
had kids," he says, adding that virtually every 
study on the subject comes to the same con¬ 
clusion: Early childbirth reduces che risk by 
about 30%. 

But epidemiologists are quick to list risk 
factors for which accurate exposure measure¬ 
ments are virtually impossible. 
Joe Fraumeni, director of the 
epidemiology and biostatis- 
tics program at che National 
Cancer Institute (NCI), 
points to radon: “When you're 
studying smoking," he says, 
“that’s easy. Just count the 
numberofcigarcttes and du¬ 
ration and packs perday. But 
something like radon, how 
do you measure exposure, 
particularly biologically rel¬ 
evant exposure that has 
taken place in the past!" 
Equally uncertain are those 
risk factors recorded only in 
human memory, such as 
consumption of coffee or di¬ 
etary fat. Ross Prentice of 
the University of Wash¬ 
ington notes, for example, 
that underweight individu¬ 
als tend to overreport fat intake on question¬ 
naires or in interviews and obese subjects 
tend to underreport it. 

Such recall bias is known to be especially 
strong, as Willett points out. among patients 
diagnosed wills rhe disease in question or 
among their next of kin. In studies of a pos¬ 
sible relationship between fat intake and 
breast cancer, for instance, says Willett, 
"people may recall their past intake of fat 
differently if they have just been diagnosed 
with breast cancer than if you pluck themout 
of a random sample, call them up out of the 
blue over the phone, and ask them what their 
past diet was." 

Recall bias, for instance, apparently nc- 
eoumsfoT the conflicting find mgs aNiot oral 
comrnccptive use and hre.isi cancer. Many 
studies have lixiked tor this association over 
the years. With case-control studies and co¬ 
hort studies. Tricllupoulos notes that case- 
conirol studies have tended to show nil asso¬ 
ciation between oral contraceptives anil 
breast cancer, while cohort studies have nor. 
Epidemiologists who have done Cohort sruJ- 
jes say the problem is m case-control studies, 
which are thrown off by recall bias—women 
who are diagnosed with breast cancer are- 
more likely to give complete information 
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about contraceptive use than women who 
don't. Hwise wlmdid cjisc- cornroi studics sa y 
the bias is in (ht cohort studies- Cohort stud¬ 
ies have to rely on impersonal questionnaires 
because they are so much larger than ease- 
control studies, and women are less likely to 
give complete and honest information than 
they are in the more incimate interviews pos¬ 
sible in case-control studies. ‘'The point,** 
says Trichopoulos, “is which do we believe.” 

lt*s not just the subjects of studies who are 
prone to bias; epidemiologic studies can be 
plagued by interviewer bias as well. The in¬ 
terviewers are rarely blinded to cases and 
controls, after all, and questionnaires, the 

traditional measuring instrument of epide¬ 
miology. are neither peer-reviewed nor pub¬ 
lished with the eventual papers. "In the labo¬ 
ratory, " as Yale University clinical epidemi¬ 
ologist Alvin Feinsccin purs ic, “you have all 
kinds of procedures for calibrating equip¬ 
ment and standardizing measurement proce¬ 
dures. In epidemiology ... it's all immensely 
prey to both the vicissitudes of human 
memory and the biases of the interview." 

Salvation from statistics? 

With confounders, biases, and measurement 
errors virtually inevitable, many epidemiolo¬ 
gists interviewed by Science say that risk-fac¬ 
tor epidemiology is increasingly straying be¬ 
yond the limits of the possible no matter how 
carefully the srudies are done “l have trouble 
imagining 3 system involving a human habit 
over a prolonged period of rime that could 
give you reliable estimates of frisle] increases 
that arc of the order of tens of percent." says 
Harvard epidemiologist Alex Walker. Even 
the sophisticated statistical techniques that 
have entered epidemiologic research over 

rhe past 20 years-tools for teasing out subtle 

effects, calculating the theo¬ 
retical effect of biases, cor¬ 
recting for possible con- 
founders, and so on-—can’t 
compensate for the limiia- 
tiuru of the data, sny.s hiostat- 
tstician Norman Breslow of 
the University of Washing¬ 
ton. Seattle. 

“In the past 30 years," 
he says, “the methodology 
has changed a lot Today 
people arc doing much more 
in the way of m.irheinnncal 
modeling of the rev.ilc.% of 
their study, fitting regres¬ 
sion equations, regression 
analysts. But rhe question re¬ 
mains; What is the funda¬ 
mental quality of the data, 
and to what extent arc there 
biases in the data that cannor 
he controlled by statistical 
analysis’ One of the dangers 
of having all these fancy 


“Authors and 
investigators are 
worried that 
there's a bias 
against negative 
studies.” 

—Marcia Angell 


mathematical techniques is 
people will think they have 
been able to control for 
things chat are inherently 
nor controllable.” 

Brestow adds that epide¬ 
miologists will commonly re¬ 
port chat they have unveiled 
a possible causal association 
between a risk factor and a 
disease because the associa¬ 
tion is ^statistically signifi¬ 
cant," meaning chat the error 
bars—the limits of a 95% 
confidence interval——do not 
include the null result, which 
Is the absence of an effect. 

But, as Breslow explains, such 
statistical “confidence” means 
considerably less chan it seems 
to. The calculation of confi¬ 
dence limits only takes into 
consideration random varia¬ 
tion in the data. It ignores 
the systematic errors, the bi¬ 
ases and confounders, that will almost in¬ 
variably overwhelm the statistical variation. 

University of California. Los Angeles 
(UCLA) epidemiologist Sander Greenland 
says most of his colleagues fail to understand 
this simple point. “What people want to do 
when they see a 95% confidence [interval]," 
he says, "is say 1 bet there's a 95% chance the 

true value is in there.’ Even if they deny it, 
you see them behaving and discussing their 
study result as though that's exactly what it 
means. There are certain conditions under 
which it's not far from the truth, but those 
conditions arc generally not satisfied in an 
epidemiologic study." 


What to believe? 

So what docs it take to 
make a study worth taking 
seriously’ Over the years, 
epidemiologists have of¬ 
fered up a variety of crite¬ 
ria. the most important of 
which arc a very strung asso¬ 
ciation herween disease and 
risk factor and a highly 
plausible biological mecha¬ 
nism. The epidemiologists 
interviewed by Science say 
they prvler ru see horh be¬ 
fore believing the latest 
study, or even the latest 
group of studies. Many re¬ 
spected epidennok «jists have 
uhi 1 shed erioncnu* results 
the past nnd snv ir is so 
easy to be fooled that it is 
Minost impossible to believe 
less-r hart-stunning results. 

Sir Richard Doll of Ox¬ 
ford University, who once 




People [may] 
think they have 
been able to 
control for things 
that are 
inherently not 
controllable." 

—Norman Breslow 
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co-authored a study errone¬ 
ously suggesting that women 
who took the anti-hyperten¬ 
sion medication reserpine 
had up to a fourfold increase 
in their risk of breast cancer, 
suggests that no single epi¬ 
demiologic study is persua¬ 
sive by itself unless the lower 
limit of its 95% confidence 
level falls above a threefold 
increased risk. Other re¬ 
searchers, such as Harvard’s 
Trichopouloi, opt for a four¬ 
fold risk increase as the 
lower limit. Trichopoulos’s 
ill-fared paper on coffee con¬ 
sumption and panctearic 
cancer had reported a 2.5- 
fold increased risk, 

“As a general rule of 
thumb." says Angell of the 
New EngLmd “we are 

looking for a relative risk of 
three or more (before accept¬ 
ing a paper for publication^ particularly tf it 
is biologically implausible or if it's a brand- 
new finding." Robert Temple, director of drug 
evaluation at the Food and Drug Adminis¬ 
tration, puts it bluntly; “My basic rule is if the 
relative risk isn’t at least three or four, forget 
it." But as John Bailar, an epidemiologist at 
McGill University and former statistical 
consultant for the NE/M, points out, there is 
no reliable way of identifying the dividing 
line. “If you see a 10 -fold relative risk and ic’p 
replicated and it’s a good study with biologi¬ 
cal backup, like we have with cigarettes anti 
lung cancer, you can draw a strong infer¬ 
ence." he says. “If it’s a 1.5 relative risk, and 
it’s only one study and even a very good one. 
you scratch your chin and say maybe." 

Some epidemiologists say that an asso¬ 
ciation with an increased risk of tens of per¬ 
cent might he believed if ic shows up consis¬ 
tently in many different studies. That's the 
rationale for meta-analysis—a technique for 
combining many ambiguous* studies t».> see 
whether they tend in the same directum 
{Science. 3 August 1990, p. 476). But when 
Science asked epidemiologists to ulcntilv 
weak associations that arc now cntvudered 
convincing because they show up rcpcutcdU . 
opinions were divided^—consistently. 

Take the question of alcohol and hrc.Kt 
cancer. More than 50 studies have been 
done, and more than 30 haw reported that 
women who drink alcohol have a 50'X, in¬ 
creased risk of breast cancer. Willett, w|\o*c 
Nurse’s Health Study was uniotvj thn>e that 
showed s positive association. calls it “highly 
probable" that alcohol increases the rhk of 
breast cancer- Among Other compelling f;u - 
tors, he says, the finding has been “repro¬ 
duced in many countries with many investi¬ 
gators controlling for lots of confounding 
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variables. «nd die a5soci3 '- ion kct P s c °mmg 
up.* IVit Greenland isn't so sure. “I’d bet 
now dtere isn't a consensus. 1 do know 
jtta from talking to people that some hold it’s 
a risk factor and others deny it." Another 
£V*ron-based epidemiologist, who prefers to 
remain anonymous, says nobody is con¬ 
vinced of the breast cancer-alcohol connec¬ 
tion‘except Walt Willett-’’ 

Another example is long-term oral con¬ 
traceptive use and breast cancer, a link thac 
has been studied for a quarter of a century. 
Thomas of the Fred Hutchinson Cancer Re¬ 
search Center says he did a meta-analysis in 
1991 end found a dozen studies show ing a 
believable association in younger women 
who w«e long-time users of oral contracep¬ 
tives. The bottom line," he says, “is it's 
taken us over 20 years of studies before some 
consistency starts to emerge. Now it’s fairly 
clear there’s a modest risk." But Noel Weiss 
of the University of Washington says he did 
a similar review of the data that left him 
unconvinced. “We don't know yet," he says. 
There is a small increased risk associated 
(with oral contraceptive use], but what that 
rep r es e nts is unclear” Mary Charleson, a 
Cornell Medical Center epidemiologist, 
calls the association "questionable." Marcia 
Angell calls it “still controversial.” 

Consistency has a catch, after all, explains 
David Sackett ofOxford University. Ic is per¬ 
suasive only if the studies use different archi¬ 
tectures, methodologies, and subject groups 
and still come up wich the same results. If the 
studies have the same design and "if there’s 
an inherent bias," be explains, “ii wouldn't 
make anydifference how many times it's rep¬ 
licated. Bias times 12 is still bias." What's 
more, the epidemiologists interviewed by 
Science point out that an apparently consis¬ 
tent body of published reports showing a 
positive association between a risk factor and 
a disease may leave out other, negative find¬ 
ings that never saw the light of day, 

‘Authors and investigators arc worried 
tliat there's a bias against negative studies,' 1 
and that they will not be able to gcr them 
published in the better journals, if at ail, says 
Angell of rheNE/M. “Ami so they’ll try very 
hard toconven what is essentially a negative 
study into a positive study by hanging on to 
very, very small risks or seizing on one posi¬ 
tive aspect of a study that is by and large 
negative." Or, as one National Institute of 
environmental Health Sciences researcher 
puts ir, asking for anonymity, “Investigators 
who find an effect pet support, and invest iga- 
tors whodon’r find an effect don’t pet support. 
When times are rough it becomes extremely 
difficult for investigators to be objective.” 

When asked why they so willingly publish 
inconclusive research, epidemiologists say 
• hey have an obligation to make the data 
public and fustify the years of work. They also 
•trgue that if the link is real, the public healcb 


SAtrr .*£*&*£. 


Special news Report 


effect may be so dramatic that it would be 
irresponsible not to publish it. The Univer¬ 
sity of North Carolina's Savicz, for instance, 
who recently claimed a possible hnk between 
EMF exposure and a tens of percent increase 
in the nsk of breast cancer, says: “This is 
minute.... But you could make an argument 
that even if this evidence is 1000-fold less 
than for jan EMF-leukemia link], it is still 
more important, because the disease is 1000- 
fold more prevalent" 

One of the more pervasive arguments for 
publishing weak effects. Rothman adds, is 
that any real effect may be stronger chan the 
reported one. Any mismeasurement of expo¬ 
sure. so the argument goes, will only serve to 
reduce the observed size of me association. 
Once researchers ieam how to measure ex¬ 
posure correctly, in other words, the actual 

association will cum out to be 
bigger—and thus more criti¬ 
cal to public health, That was 
the case in studies of steel¬ 
workers and lung cancer de¬ 
cades ago, says Robins. Early 
studies saw only a weak asso¬ 
ciation, but once researchers 
homed in 1 on coke-oven 
workers, the group most ex¬ 
posed to the carcinogens, the 
relative nsk shot up. None 
of the epidemiologists who 
spoke to Science could recall 
any more recent parallels, 
however. 

An unholy alliance 
There would be few draw¬ 
backs to publishing weak, 
uncertain associations if epi¬ 
demiologists operated in a 
vacuum, wrote Brian Mac- 
Mahon, professor emeritus of 
epidemiology at Harvard, in 
an April 1994 editorial in rhe 
Journal of the National Cancer Institute. Bur 
they do not. he said. “And, however cau¬ 
tiously the investigator may report lur. con¬ 
clusions and stress the need for further evalu¬ 
ation,'' he added, "much of the press mil pay 
little heed to such cautions. ... By the rime 
the information reaches the public mind, via 
print or screen, the tentative suggestion is 
likely to he interpreted us a fact." 

This is ivhar Line epidemiologist calls the 
“unholy alliance between epidemiol. njy. the 
journals, and the lay press. The first one or 
rwo papers about a suspected a-ooci.ilion 
“spring into the general public cunscioiiMiess 
in way that doer not hapj-ien in all! other 
field of scicncihc endeavor," says Harvard's 
Walker And once a possible link is in the 
public eye, ir can be virtually u'tipo-sil'le to 
discredit. As far a; scientists were concerned, 
for instance, a 19?) epidemiologic -tu.lv pot 
to rest a suggestion that saccharine can Cause 


bladder cancer—one of rhe few cases in 
which epidemiology had managed to put an 
end to a suspected association. Yet 14 years 
later, television advertisements for Nucra- 
Sweet, which contains the artificial sweet¬ 
ener aspartame, still tom it as the sweetener 
that docs not have saccharine. 

Epidemiologists themselves are at a loss as 
to how to curb the “anxiety of the week" 
syndrome. Many, like Rothman, simply ar¬ 
gue that risk factor epidemiology is a young 
science that will take time to mature. Ochets, 
like Robins, suggest that barring a major 
breakthrough in che methodological tools of 
epidemiology, maturity will be hard to come 
by. The pressures to publish inconclusive re¬ 
sults and the eagerness of the press to publi¬ 
cize them, he and others say, mean that the 
anxiety pendulum, like Foucault's, will con¬ 
tinue to swing indefinitely 
(see box on p. 165). 

The FDA's Temple does 
make one positive sugges¬ 
tion: Although risk-factor 
epidemiology will never be 
as sharp a tool as random¬ 
ized clinical trials, epidemi¬ 
ologists could still benefit by 
adopting some of the scien¬ 
tific practices of those stud¬ 
ies. “The gteat thing about a 
clinical control trial." he 
says, “is that, within limits, 
you don’t have to believe 
anybody or trust anybody. 
The planning for a clinical 
control trial is prospective; 
they've written the protocol 
before they've done the 
study, and any deviation 
that you introduce later is 
completely visible." While 
agencies like the NCI do in¬ 
sist on seeing study proto¬ 
cols in risk-factor epidemi¬ 
ology prospectively, this is still not standarj 
procedure throughout the field Without ir. 
says Temple, “you always wonder bow- 
many ways they cut the data. It's very h.itil 
to he reassured, because there are no rules 
(or doing it." 

In the meantime, UCLA's Greenland has 
one piece of advice to offer what he calls his 
' “iiiiim sensible, IcveThcwdcJ, estimable col¬ 
leagues." Remember, he says, "there is noth¬ 
ing sinful nlxuit going out and getting evi¬ 
dence, like asking people how 1 much dn you 
drink and checking hfeiist cancer records. 
There's nothing sinful alxiut Seeing if that 
evidence correlates. There's nothing sinful 
alvuit checking for confounding variables. 
The sin comes in believing a causal hvporh- 
esis is true because your study came up with ,i 
positive result, or believing the opposite be¬ 
cause your study was negative.*’ 

—Gary Taubes 



‘The sin comes in 
believing a causal 
hypothesis is 
true because 
your study came 
up with a positive 
result.” 

—Sander Greenland 
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